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The Variance of Intraclass Correlations in Three and Four Level Models 

Intraclass correlations play important roles in genetics, epidemiology, 
psychometrics, and the design of social and educational experiments. In multistage 
sampling models and multilevel statistical models, they play a major role in quantifying 
the amount of clustering inherent in the data. The intraclass correlation was introduced 
by Fisher (1925) who offered an estimator of the intraclass correlation, derived its 
variance in balanced experiments, and discovered a variance stabilizing transformation. 
Estimators of the intraclass correlation from unbalanced designs and estimators based on 
slightly different principles have subsequently been introduced. See Donner (1986) for a 
survey of the literature on point and interval estimation of the intraclass correlation. 

Intraclass correlations are important input parameters used for power 
computations and the computation of optimal sample allocations among levels in the 
design of multilevel randomized experiments (see Raudenbush, 1997; Kostantopoulos, 

2009) , and for computation of effect size estimates and their variances (see Hedges, 2007, 

2010) . Consequently there has been considerable interest in the estimation of intraclass 
correlations from sample surveys using multistage samples to estimate intraclass 
correlations (e.g., Hedges & Hedberg, 2007). Such studies typically fit unconditional 
multilevel models to the survey data to estimate variance components at each level of the 
sampling design. Other studies use datasets that were assembled in the course of carrying 
out randomized experiments (Bloom et ah, 2007). While the surveys often have large 
total sample sizes, the number of sampled units at some levels (typically the higher 
levels) may not be large enough make negligible the sampling uncertainty of estimates of 
variance components and functions of variance components (such as intraclass 
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correlations). Even experiments that are nonnatively large (that is large for experiments) 
typically involve much smaller sets of schools than national surveys (typically less than 
100 schools). Consequently, for either type of study designed to provide reference values 
of intraclass correlations, it is important to provide some assessment of the uncertainty of 
the estimates. However because the sample sizes in surveys are actually large, estimates 
of sampling uncertainty based on large sample methods should be accurate enough to 
give this guidance. 

Although intraclass correlation was originally introduced in relation to two level 
sampling models, the concept extends naturally to sampling models with three or more 
levels. The intraclass correlation concept in cases of three and four level sampling 
models is of great interest in the design and analysis of experiments in education (Hedges 
and Rhoads, 2010; Konstantopoulos, 2008ab, 2009). Three and four level experimental 
designs are of more than theoretical interest. A survey of recent educational experiments 
revealed that the most common designs actually involve four levels of sampling 
(Spybrook and Raudenbush, 2009). The literature on estimation of intraclass 
correlations, however, has largely been restricted to the case of two level models. An 
exception is a paper by Raykov (2010) that derived the large sample standard error for 
one of the intraclass correlations in the three level hierarchical linear model. We became 
interested in the problem of quantifying the uncertainty of estimates of intraclass 
correlations in three and four level sampling models as part of a project to develop 
estimates of three and four level intraclass correlations from survey data to provide 
guidance for planning randomized experiments in education. 
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The object of this paper is to provide simple estimates of the sampling variances 
and covariances of intraclass correlation estimates derived from multilevel statistical 
analyses of survey data. For example, suppose that we estimate the variance components 
associated with three level fully unconditional model (no covariates at any level of the 
model). Let of, cry, and a 3 , be the variance components at the levels 1, 2, and 3 and let 
s 2 , s 2 , and S 3 be their maximum likelihood estimates. The large sample variances of the 
intraclass correlation estimates depend on the variances of the variance component 
estimates and on their covariances (when there are three or more levels). 

Note that the residuals at different levels are usually assumed to be independent. 
However this does not mean that the estimates of residual variances from different levels 
are independent. In fact, in balanced designs where closed form expressions for the 
covariances are available, it is easy to see that the estimates are not independent (see 
Casella, McCollough, and Searle, 1992) 

Statistical software for multilevel analyses (e.g., HLM) does not always provide 
estimates of the covariances of variance components at different levels, however. That is, 
while HLM and other programs provide the variances of s 2 , s 2 , and sf, they do not 
provide the covariance of s 2 and S3 2 . Therefore we have emphasized providing 
expressions for the standard errors of the quantities of interest that do not require 
knowledge of the covariances between variance component estimates at different levels. 

In realistic situations involving large scale survey data, the level 1 variance 
component is essentially kn own because there are so many level 1 units, so that s 2 ~ of. 
That is, the variance v/ of s 2 is essentially zero. Let v, be the variance (squares of the 
standard errors of) the variance component estimates sf for j > 1. STATA, HLM, or SAS 
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Proc Mixed provide the standard errors of variance component estimates at level 2 or 
higher, so these (and therefore the corresponding vj) are available even in unbalanced 
cases. 

Two Level Model 

In a two level model there are two variance components of and o 2 , which are 
estimated by s 2 and s 2 , respectively, where s 2 ~ of. Let vj and v 2 be the variances of 
sf and s 2 . The condition s 2 ~ o 2 implies that v; = 0. Let in denote the number of 
clusters (level 2 units) and denote the number of level 2 units in the i th level 2 unit. 
When the design is balanced, ni=' = n m = n. The interclass correlation in the two level 
model is 


P 


o: 


2 2 

07 + <7; 


which is estimated by 


2 . 2 

+^ 2 


Then the large sample variance of r, the estimate of p in a balanced design is 


(1-P) 2 


< 7 t 


( 1 ) 


where v 2 is the variance of s 2 , the sample estimate of the level 2 variance component and 
ot = o 2 + o 2 2 is the total variance. The sample estimate of the variance of r is obtained 
by replacing all of the parameters in (1) by their samples estimates, that is the estimate of 
the variance of r is 

( 1 -r ) 2 v 2 


h 2 +* 2 2 f 
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The standard error of r is just the square root of its estimated variance. 

Expression (1) does not appear to be similar to conventional expressions for 
estimates of the variance of intraclass correlations in balanced designs. For example, the 
expression given by Fisher (1925) for the large sample variance of the estimator of p is 
(in our notation) 

l{\-p) 2 \} + {n-\)pf 
n(n-l)(m-l) 

Donner and Koval (1980) gave an expression for the large sample variance of r computed 
from unbalanced designs as 


2N(l-pf 


m i 0 m 

N Y\} + { n i- x ) P 2 ]/\} + { n i- x ) p] -P 2 (« ; -!)/[! + («, -!)p] 

i =1 L i =1 


-|2 ’ 


( 3 ) 


where N is the total sample size. Note that (3) reduces to (2) when m = " = n m = n. 


Noting that the variance of S 2 in a balanced design is 


2 , 2 
(Ty -\-fV72 


{m— l) 


2 aj [l + (/7 — \)p 

n 2 (m — l) 


we see that (1) differs from (2) only in that the implicit n tenn in the denominator of (1) 
corresponds to an n(n - 1) tenn in the denominator of (2), that is they differ only in terms 
of order n 2 , which implies that they are equivalent in large samples (that is, as « —> oo). 

Three Level Model 


In the three level model, there are three variance components, one associated with 
each level of the model. Denote the number of level 3 units (clusters) by m, the number 
of level 2 units (subclusters) within the /' th level 3 unit by p h and the number of level 1 
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units (individuals) within the j th level 2 unit within the 7 th level 3 unit by ny. For 
example, if level 3 units are schools, level 2 units are classrooms, and level 1 units are 
individuals, then m denotes the number of schools, pi is the number of classrooms in the 
7 th school, and ny is number of students within the/ h classroom of the 7 th school. If the 
design is balanced, then pi = "' =p m and ny = n, i = 1, 1, ...,p. Let a 2 , oi, and 

oi, be the level 1, 2, and 3 variance components, and let si, si, and .s'/, be their 
maximum likelihood estimates. In realistic situations, the level 1 variance component is 
essentially known because there are so many level 1 units, so that si ~ o 2 . Let vj and v$ 
be the variances (squares of the standard errors of) the variance component estimates si 
and si. 


STATA provides the standard errors of variance component estimates, so these 
(and therefore v 2 and v?) are available even in unbalanced cases. In the balanced case, it is 
easy to show that covariance between si and si is -v 2 /p. Using this covariance, we can 
get the (large sample) variance of the intraclass correlation estimates. 

Define the level 2 intraclass correlation as 


<7 ; _ a; 

Pi ? 1 1 

(7~ + <J 2 + <T 3 <7 t 


and its estimate as 


r = ^2 _ ^2_ 

2 .sf+s 2 2 +s 3 2 s 2 t 


Define the level 3 intraclass correlation as 


A 


(t: 


a 


___ 3 _ 

2 2 2 2 

<7[ + <J 2 + <J 3 Oj 


and its estimate as 
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3 „2 . 2 . „2 2 ‘ 

+ 5 2 + S^ S T 

The expressions for the large sample variances of intraclass correlations in the 
unbalanced three level model are quite complicated, even asymptotically. A quick 
approximation can be derived by assuming that the covariances of the maximum 
likelihood estimates of the variance components in the unbalanced case are the same as in 
the balanced case. In a balanced design, the number p of level 2 units per level 3 unit is a 
constant. In an unbalanced design, use an average value (such as the mean or harmonic 
mean) in place of p in the fonnulas below. 

A straightforward derivation leads to the following variances assuming that the 
covariances between s 2 and s 2 are the same as in a balanced design. The variance of r 2 


^0 Pi ) + 2/r(l Pi) v 2 pfvj 

4 ' 4 ’ 

p(7j, (7j 

the variance of r? is 

\_PPi T2/? 3 (l — Pi )Jv 2 (l -p 3 ) V 3 

4 I 4 

pO T Oj 

and the covariance between r 2 and r? is 

-[pp 3 (l-p 2 ) + p 2 p 3 + (1 ■- p 2 )(1 ■- p 3 )] p 2 (1 - p 3 )v 3 

4 4 

pCT T 0 T 


( 4 ) 


( 5 ) 


( 6 ) 


Because you have the variance component estimates s 2 , s 2 , and .S 3 2 , you can compute r 2 
r 3 , and st = .s'/^+ s 2 + S 3 2 . Then, because we also have the variances v 2 and v 3 of the 
variance component estimates s 2 ', and S 3 2 , we can compute estimates of the variances and 
covariance above by substituting the sample estimates r 2 , r 3 , and sf in place of p 2 , p 3 , and 
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ot in (4), (5), and (6). The standard errors of r 2 and r 3 are just the square roots of their 
estimated variances. 

When the design is unbalanced, the covariance between maximum likelihood 
estimates of the variance components s 2 and s 3 is more complex. Using results from 
Searle (1970), the covariance c 23 between s 2 and s 3 is 



m 

I«-/(l+^3 2 ) 


1=1 

where a t is given by 


a ,=Yj n nl{ n ii a2 2 +a i)- 

./=' 

Note that when the design is balanced so that p,= p and n,y = n for all i and j, then this 
covariance reduces to —v 2 /p as it should. 

Using this covariance for unbalanced designs, the variance of r 2 is 

i}~ Pi) v 2 , P 2 v i 2p 2 (l —/k)c 23 {q\ 

-5 + —i-4- W 

(Tj’ (Tj 1 


and the variance of r 3 is 


/UU 0 Pi) v i (1 Pi) c n 

4 ' 4 4 


( 9 ) 


Calculations using (4) show that, unless there is extreme imbalance, the value of the 
variances of r 2 and r 3 computed from (8) and (9), respectively are remarkably similar to 
those obtained by using the mean of the p t in (4) and (5) respectively. 


Four Level Model 
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In the four level model, there are four variance components, one associated with 
each level of the model. Let of, of', of, and of, be the level 1, 2, 3, and 4 variance 
components, and let sf, sf, sf, and sf, be their maximum likelihood estimates. Denote 
the number of level 4 units (clusters) by m, the number of level 3 units (subclusters) 
within each level 4 unit by q, the number of level 2 units (sub-subclusters) within each 
level 3 unit by p, and the number of level 1 units (individuals) within each level 2 unit by 
n. For example, if level 4 units are school districts, the level 3 units are schools, the 
level 2 units are classrooms, and level 1 units are individuals, then m denotes the number 
of school districts, q is the number of schools per district, p is the number of classrooms 
per school, and n is number of students per classroom. Note that we assume the design is 
balanced. In realistic situations, the level 1 variance component is essentially known 
because there are so many level 1 units, so that s 2 ~ of. Let v?, vj, and v 4 be the 
variances (squares of the standard errors of) the variance component estimates sf, S3 2 , 
and sf. Standard software provides the standard errors of variance component estimates, 
so these (and therefore V2, V3, and v 4 ) are available even in unbalanced cases. In the 
balanced case, it is easy to show that covariance between sf and S 3 is -vy/?, where p is 
the number of level 2 units per level 3 unit, the covariance between S3 2 and sf is 





Standard errors of intraclass correlations 11 


and its estimate as 


iS > 2 ^2 

sf + si + si + si si 


Define the level 3 intraclass correlation as 


Pi 2 2 2 2 2 

<7, + C7 j + <7 3 + (J 4 <7j 


and its estimate as 


3 ^2.2.2. 2 2 

+ s 2 + s 3 + £4 .sy 


Define the level 4 intraclass correlation as 


2 9 2 T 2 

(J; + (j; + <7 3 + <7 4 " <7‘ 


and its estimate as 


r s * _ *4 

4 v, 2 + s] + si + si si 


Assuming that the covariance of the variance components is the same as in the 
balanced case, the large sample variance of the estimators r?, /;?, and r 4 of the intraclass 
correlations can be obtained in terms of the variances V 2 , V 3 , and v 4 . 

The variance of r 2 is 


VP 2 ( 1 - Pif +2 qPP 2 { 1 - Pi) + 2 P 2 2 v 2 (q~2)p 


2 4 

qp a T 


- 2 )p 2 i v i P 2 2 v 4 

4 4 

qcj y (7 y 


The variance of r? is 


\qp 2 p 2 +2(^p—1)/» 3 (i-/h)]v 2 q(}-pi) + 2 Pi{ l -pi) v i 


2 4 

qp g t 
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The variance of r 4 is 


\qp[ p~ 2 )pI- ip, (i -/Q]v 2 | \qpl + 2 p 4 (i- p 4 )] v 3 | (1 -p 4 ) 2 


2 4 

qp o T 


4 

qo T 


( 12 ) 


The covariance between r 2 and r 3 is 

\-qP~PlP 3 _ W [PlP-i + PlPi ) - P2P3 + P2P3 ] V 2 U/P2P3 - P 2 P 2 + P2P2 ] V 3 P 2 Pi V 4 


2 4 

qp a T 


4 

qo T 


- + - 


( 13 ) 


the covariance between r? and is 

\-qp-p 2 P 4 + qp (p 2 p 4 - p 2 p 4 ) + p 2 p A ~ p 2 p 4 ] v 2 [g-p 2 p 4 - p 2 p 4 + p 2 p 4 ] v 3 p 2 p 4 v 4 


2 4 

qp a T 


- + ■ 


4 

qa T 


, ( 14 ) 


and the covariance between r? and r 4 is 

\qp-p 2 p 4 + qp (p : JJ 4 - p 3 p 4 ) + p-,p 4 - p j) 4 ] v 2 [qp,p 4 + p 3 p 4 + p 3 p 4 ] v 3 p 3 p 4 v 4 


2 4 

qp o T 


4 

qo T 


, ( 15 ) 


where p 2 =1 -p 2 ,p 3 = 1 -p 3 , and p 4 = \-p 4 . 

As in the three level case, we can compute estimates of the variances and 
covariances above by substituting the sample estimates r 2 , r 3 , r 4 , and sf in place of p 2 , p 3 , 
p 4 , and ot in (10) through (15). The standard errors of r 2 , r 3 , and r 4 are just the square 
roots of their estimated variances. 

Examples 

For the examples presented in this paper we utilize a sample of the reading results 
from the 2009-2010 Kentucky Core Content Test, which is the state’s test given to 
students in the spring of the academic year (Kentucky Department of Education, 2012). 
Our data includes reading scores from 46,849 fifth graders. These test scores are spread 
across 173 districts, 715 schools, and 2,142 teachers. The hannonic mean of the number 
of schools per district was 1.81 (arithmetic mean was 4.13), the harmonic mean number 
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of teachers per district was 4.12 (arithmetic mean was 12.38), and the harmonic mean 
number of teachers per school was 2.04 (arithmetic mean was 3.00). The arithmetic 
average of 22 students per teacher (harmonic mean was about 8 students per teacher). 

While the official evaluation of students includes both open ended responses as 
well as multiple choice responses, we focused only on the multiple choice portion of the 
test. This part of the test includes 39 items that are scored as correct or incorrect, with the 
final score representing the number of correct responses. In our data, the test scores 
ranged from 0 to 33, with a mean of 27.01 and a variance of 27.51. 

The results of our example analyses are presented in Table 1. For the two level 
example, we fit a mixed effect model where we nested students (level 1) in schools (level 
2). For the three level model, we fit a mixed model where we nested students (level 1) in 
teachers (level 2), and teachers in schools (level 3). In the four level model, our mixed 
effect model nests students (level 1) in teachers (level 2), teachers in schools (level 3), 
and schools in districts (level 4). In order to keep our example numbers tractable, we did 
not standardize our outcome, nor did we include any covariates in our models. 

Two Level Models 

Fitting a two level model to the Kentucky dataset, with students nested within 
schools, the total variance is estimated to be st = 27.650, the level 2 variance component 
is S 2 = 2.409, and the variance of the level 2 variance component is V 2 = 0.024. With 
these parameters, we calculate the estimate of the school level intraclass correlation to be 
r = 0.087. We then calculate the variance of r as 


(1-0.087) 2 0.024 _ 0.020 
27.650 2 ~ 764.523 


0.00003, 


var(r) 
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which implies a standard error of^O.00003= 0.005. Donner’s formula gave 
approximately the same result for the standard error, also rounding to 0.005. 

Three Level Models 

We return to the Kentucky data for our example of a three level fully 
unconditional model, where students are the level 1 units, teachers define the second 
level, and schools define the third level. The outcome variable is again the raw reading 
scores. The parameter estimates from the model are s? = 3.190, = 1.557, St = 28.582, 

v? = 0.040, V 3 = 0.030, r? = 0.112, and r? = 0.054. The harmonic mean of the number of 
teachers per school is p = 2.042. Using the formulas for the balanced case with p = 

2.042, we calculate the variance of r? as 


Var(r 2 ) 


2.042(1-0.112) 2 + 2(0.112)(1-0.112) 0.040 Q. 112 2 (0.030) 


2.042 (28.582 2 ) 


28.582 2 


0.072 + 0.0004 = a00004 + a 0000005 = 0.0000405 
1668.173 816.931 


which implies a standard error ofiVo.0000405 = 0.006. 

Using the fonnulas for the balanced case with p = 2.042, we calculate the variance 


of r? as 


Var(r 3 ) = 


(2.042)(0.054 2 ) + 2(0.054)(l-0.054)j0.040 (1-0.054) 2 0.030 


0.004 


- + - 


2.042 (28.582 2 ) 
0.027 


28.582 2 


1668.173 816.931 


= 0.000002 + 0.00003 = 0.000032 


which implies a standard error of Vo.000032 = 0.006. 


Four Level Models 
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Using the Kentucky data for our four level model example, we fit a fully 
unconditional mixed model with teachers defining the second level, schools defining the 
third level, and school districts defining the fourth level. We again used the raw reading 
scores. The parameters from the model are s 2 = 3.160, s 3 = 0.957, s 4 = 0.314, sj = 
28.265, v 2 = 0.039, v 5 = 0.021, v 4 = 0.008, r 2 = 0.112, r 3 = 0.034, and r 4 = 0.011. The 
hannonic mean of the number of teachers per school is p = 2.042, and the harmonic mean 
for the number of schools per district is q = 1.818. Using the fonnulas for the balanced 
case, we calculate the variance of r? as 


Var(r 2 ) = 


1.818 (2.042 2 )(l — 0.112) 2 + 2 (1.818) (2.042)0.112(1-0.112)+ 2 (0.112 2 ) 


0.039 


+ 


1.818(2.042 2 )28.265 3 
(1.818 — 2)(0.112 2 ) 0.021 (0.112 2 )0.008 


1.818 (28.265 2 ) 


28.265 2 


0.263 -0.00005 

~~ 6056.244 1452.419 

= 0.00004007 


0.0001 

798.910 


0.00004 + (-0.00000003) + 0.0000001 


which implies a standard error of \J0 .00004007 = 0.006. 
The variance of r 3 is calculated as 


Var(r 3 ): 


1.818(2.042 2 )0.034 2 + 2 (l.818 (2.042)-l) 0.034 (1-0.034) 


0.039 


+ 


1.818(2.042 2 )28.265 2 

1.818(1 -0.034) 2 + 2(0.034)(1-0.034 ) 0.02 1 0.034 2 (0.008) 


0.007 


- + ■ 


1.818 (28.265 2 ) 
0.037 0.00001 


- + - 


6056.244 1452.419 798.910 


28.265 2 

: 0.000001 + 0.00003 + 0.00000001 


which implies a standard error of^O.00003101 = 0.006. 


The variance of r 4 is calculated as 
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Var (r 4 ) = 


1.818 (2.042) (2.042 — 2)(0.011 2 ) — 2(0.01 l)(l — 0.011) 


0.039 


1.818(2.042 2 )28.265 2 


+ 


[l.818(0.011 2 ) +2(0.01 l)(l-0.011) 


1.818 (28.265 2 ) 


0.021^ (1 — 0.011) 2 0.008 


28.265 


"O' 00 1 - + Q ' QQQ5 + Q,QQ8 = -0.0000002+0.0000003+0.00001 


6056.244 1452.419 798.910 


which implies a standard error of^O. 0000101 = 0.003. 


ICCVAR Software for Stata 


We have developed software to perform these calculations in Stata. Once the user 
downloads and installs the ICCVAR.ado program, performing these calculations is quite 
simple. The first step is to run the mixed model as you normally do in Stata. Then, 
immediately after the mixed model is estimated, simply type in “iccvar” into the 
command prompt to estimate the ICCs and variances, just as you would any other Stata 
“post-estimation” command. The program then uses the variance components and 
variances stored by the estimation command to perform the calculations. The ICCVAR 
program automatically detects the number of levels used in the previous mixed model. 
When a three level model is specified, you can enter an optional “unb” command to use 
the unbalanced formulas. Figure 1 provides an example session using the program. This 
program also stores the matrix of intraclass correlations and the variance-covariance 
matrix in the results memory. This SATA code can be downloaded from 
http://www.ipr.northwestern.edu/qcenter/iccvar.html . 


Conclusions 
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We have provided formulas for the large sample variances and covariances of 
estimates of intraclass correlations in three and four level models. These expressions are 
suitable for computing standard errors for estimates of three and four level intraclass 
correlations when the sampling designs are balanced or nearly so. These should be useful 
in providing some estimates of sampling uncertainty for analyses designed to yield 
reference values of intraclass correlations based on large scale data collections. 
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Figure 1: Example Stata session using ICCVAR 

. quietly : xtmixed reading || schno :, var 
. iccvar 

Intraclass Correlation Estimates 



ICC 

Std. Err. 

[95% Conf. 

Interval] 

schno 

0.08714 

0.00507 

0.07720 

0.09707 

. quietly : xtmixed reading || schno : || 

| readingid 

: , var 

. iccvar 





Intraclass Correlation Estimates 



Harmonic Mean 

of Level 2 

Units per Level 

3 Unit = 

2.042 


ICC 

Std. Err. 

[95% Conf. 

Interval] 

schno 

readingid 

0.05447 

0.11162 

0.00593 

0.00662 

0.04284 

0.09864 

0.06610 

0.12460 

. iccvar, unb 





Intraclass Correlation Estimates 



Harmonic Mean 

of Level 2 

Units per Level 

3 Unit 

2.042 


ICC 

Std. Err. 

[95% Conf. 

Interval] 

schno 

readingid 

0.05447 

0.11162 

0.00586 

0.00650 

0.04298 

0.09888 

0.06596 

0.12436 

. quietly : xtmixed reading || leaid : I 

schno : I I 

readingid :, 

. iccvar 





Intraclass Correlation Estimates 



Harmonic Mean 

Harmonic Mean 

of Level 2 

of Level 3 

Units per Level 
Units per Level 

3 Unit = 

4 Unit = 

2.042 

1.818 


ICC 

Std. Err. 

[95% Conf. 

Interval] 

leaid 

schno 

readingid 

0.01111 

0.03385 

0.11179 

0.00312 

0.00512 

0.00658 

0.00500 

0.02381 

0.09889 

0.01722 

0.04389 

0.12470 
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Table 1: Example Results 


Two Level Model Three Level Model Lour Level Model 



Estimate 

Variance 

Estimate 

Variance 

Estimate 

Variance 

Lixed Effect 







Intercept 

26.931 

0.004 

26.665 

0.005 

26.872 

0.007 

Random Effects 







District 

— 

— 

— 

— 

0.314 

0.008 

School 

2.409 

0.024 

1.557 

0.030 

0.957 

0.021 

Teacher 

— 

— 

3.190 

0.040 

3.160 

0.039 

Student 

25.241 

0.000 a 

23.835 

0.000 a 

23.834 

0.000 a 

Total Variance 

27.650 


28.582 


28.265 


ICCS 







District 

— 

— 

— 

— 

0.011 

0.0000101 

School 

0.087 

0.00003 

0.054 

0.000032 

0.034 

0.00003101 

Teacher 

— 

— 

0.112 

0.0000405 

0.112 

0.00004007 


Notes: N= 46,849 students in 2,142 teachers in 715 schools in 173 districts. The harmonic 
mean of teachers per school is 2.042, the harmonic mean number of schools per district is 
1.818, a: we assume this variance to be 0.000 
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Appendix 

This appendix describes the multilevel models on which the results of this paper 
are based and the method used to derive those results. 

Two Design 

Suppose that m clusters of size n. Let Yy be they' th observation in the 7 th cluster. 
Then the level 1 (individual level) model is 

Yij = Poi + £ij ,7=1,..., m; 7=1, ...n 

where [>m is mean of the 7 th cluster and the sy are independently normally distributed with 
mean 0 and variance of. The level 2 (cluster level) model is 
Poi = yOO+ tlOi, 7 = 1 , ..., 777, 

where yoo is the grand mean and the rjoi are independently normally distributed with mean 
0 and variance of. 

We use the delta method to obtain the results of the paper (see, e.g., Rao, 1973, pp 
385-387), which implies that if a random variable 7 is a function f(7) of another random 
variable T which has variance v, then the variance of Y in large samples is (di(0)ldT) 2 v, 
where 6 = E(7). Starting with sp, we note that r is a function of sp which is random and 
sf ~ oi , which is a constant. Therefore r = ffsy) = sfK sf + sj ), and dl'/dsj = 

(1 - r)/s 2 2 , which implies (1). 

Three Level Design 

Suppose that m clusters, each with p subclusters of size n. Let Yyk be the k th 
observation in / h subcluster of the 7 th cluster. Then the level 1 (individual level) model is 

Yijk = Poij + Sijk ,7=1, . .., m;j = 1 , ... p; k = 1 , ...n 
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where Poy is mean of the j th subcluster in the 7 th cluster, and the syk are independently 
nonnally distributed with mean 0 and variance of. The level 2 (subcluster level) model 
is 

Poy = foot + rjoijj = 1, 7 = 1,..., m, 

where yoo is the mean of the 7 th cluster and the rjoy are independently nonnally distributed 
with mean 0 and variance of . The level 3 (cluster level) model is 

yooi = ft 00 + Zoi, 7 = 1 , ..., 777, 

where non is the grand mean, and the fo, are independently normally distributed with 
mean 0 and variance of 2 . 

We use the multivariate delta method to obtain the results of the paper (see, e.g., 
Rao, 1973, pp 388-389). This method uses the fact that if a 1 xp vector of random 

variables T = (7), ..., T p ) has pxp covariance matrix £, and if Y = f(T) = (fi(T), ..., 
f ? (T)) is a I x q vector of differentiable functions of T, then the covariance matrix of Y in 
large samples is given by ALA’, where A is a q x p matrix in which the element of the 7 th 

row and/ h column is (3f(0)/rJ7}), where 0 = E(T). Apply this result with q = 2 and p =2. 
Define T = (s 2 2 , S 3 2 ), note that s 2 is a constant, r 2 = fi(T) = .s'?"/( s 2 + s 2 2 + S3 2 ), and r 3 = 
f 2 (T) = .S'.r7( sf + s 2 + S 3 2 ). Noting that (3f(0)/d7}) = (1 -/:>,)/( of 2 + o 2 + o s) when 7 =j 
and (3f(0)/37)) = - pj{ of + o 2 + d 3 ) when 7 ^ j yields (4), (5), and (6). 

Four Level Design 

Suppose that m clusters, each with q subclusters in each cluster, and p sub¬ 
subclusters within each subcluster of size n. Let Yyu be the / th observation in k lh sub¬ 
subcluster of the j th subcluster of the 7 th cluster. Then the level 1 (individual level) model 
is 
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Yjjki = fioijk + Sijki ,i=l,..., m;j=\, ...p; k = 1, .. .q; l = 1, .... n 
where fioijk is mean of the k th sub-subcluster of the/ h subcluster in the /' th cluster, and the 
Sijki are independently normally distributed with mean 0 and variance op 2 . The level 2 
(sub-subcluster level) model is 

Poijk = yoij + rjoijk, i = 1, rn;j= 1, ...,p;k= 1 , q, 
where yotj is the mean of the j th subcluster in the 7 th cluster and the rjoijk are independently 
normally distributed with mean 0 and variance oi 2 . The level 3 (subcluster level) model 
is 

yOij — 7t0i T Q0ij,j ~ 1, ■ P, i ~ 1, •••, 777, 

where 7 xm is the mean of the 7 th cluster and the £oij are independently normally distributed 
with mean 0 and variance of. The level 4 (cluster level) model is 

Xoi = 00 + c oi , i = 1, tn, 

where do is the grand mean, and the Coi are independently normally distributed with mean 
0 and variance oi. 

Use the multivariate delta method again with q = 3 and p = 3. Define the 1x3 

vector T = (s/, S 3 2 , s/) note that s 2 is a constant, r 2 = fi(T) = s 2 2 l{ sf + s 2 + s$ + s/), 
and ri = f 2 (T) = s 2 2 /( sf + si + S 3 2 + s/), and 7 -4 = f 2 (T) = s//( sf + s 2 + S 3 + s/). 
Noting that Of(0)/d7}) = (1 - pi) K of + o 2 + o 3 + o/) when i = j and (3f,(0)/d7)) = 
-pA a > 2 + a 2 + o 2 + <7 4 ) when 7 f j yields (10) through (15). 



